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Abstract: It has now been 20 years since the seminal work by Finlayson et al. on the use 
of spectral sharpening of sensors to achieve diagonal color constancy. Spectral sharpening is 
still used today by numerous researchers for different goals unrelated to the original goal 
of diagonal color constancy e.g., multispectral processing, shadow removal, location of 
unique hues. This paper reviews the idea of spectral sharpening through the lens of what 
is known today in color constancy, describes the different methods used for obtaining a set 
of sharpening sensors and presents an overview of the many different uses that have been 
found for spectral sharpening over the years. 
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1. Introduction 

Our visual system has a striking ability in allowing us to deal with color. However, we are far from 
fully understanding its behavior. To gain an insight into how our visual system works, some assumptions 
are often made: first, it is assumed that there is a single illuminant in the scene which is spatially uniform, 
and second, it is assumed that objects are flat, coplanar, and Lambertian, i.e., their reflectances are diffuse 
and independent from the angle of view. 

Following these assumptions light energy reaching our eye depends on the spectral power distribution 
of the illuminant ( E(X), where A spans the visible spectrum) and the spectral reflectance distribution of 
the object we are looking at (R(Xj). This information is called the color signal and is written as 



C(A) = R(X)E(X) 



(1) 
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This color signal is weighted along the visible spectrum to with the sensitivities of our cone-cells 
(which peak in the long, medium and short wavelengths of the visible range, and are denoted by 
s(X),m(X),l(X) respectively) to obtain the L,M,S color space coordinates of the signal 



{L,M,S}= C{X){l,m,s}{\)dX 



(2) 



From this equation, we can see that when looking at a white piece of paper in sunset the values 
captured by our eye are reddish as a result of the illumination. In contrast, when looking at a white 
piece of paper on a cloudy day these values are bluish. However, we perceive the piece of paper as 
approximately white in both cases. This property of our visual system is called color constancy. 

1.1. Human Color Constancy 

Color constancy is usually defined as the effect whereby the perceived or apparent color of a surface 
remains constant despite changes in intensity and spectral composition of the illumination [1]. An 
example is shown in Figure 1, where we are able to perceive the t- shirt of the man in the right as yellow; 
however, if we isolate the t-shirt we perceive it as green. Some reviews on color constancy have been 
published recently [1-3]. 

Figure 1. Example of color constancy. We are able to perceive the t-shirt of the man in the 
right as yellow, but, when looking at it in isolation the color of the t-shirt appears green. 




von Kries [4] in the XlXth century hypothesized that a compensation (or gain) normalization was 
performed individually within each photoreceptor. Mathematically, we may write 
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where the subscript in represents the values captured by the eye. 



Sensors 2014, 14 



3967 



This model is called von Kries adaptation. The idea of an individual gain for each photoreceptor was 
adopted early in computer vision and the gains were computed based on the scene, e.g., assuming that 
the scene had a mean value of grey [5], or that a white-patch was presented on the scene [6]. von Kries 
adaptation has been shown to provide a reasonable approximation of how we perceive scenes composed 
of natural reflectances and illuminant spectra [3]. In a contemporany version of the model, it is stated 
that cone-signals interact and the gain in each channel is influenced by the other channels [7] 
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(4) 



Even though this model can predict the data very well in natural environments, there is still no 
agreement in the literature as to how the gain values are computed from the image statistics of the 
stimulus [3]. Furthermore, this model performs poorly when natural reflectances and illuminants are not 
present in the stimulus. 

For this reason, further research on the neural mechanisms that underline color constancy has been 
conducted over the years [1]. Different parts of the brain have been shown to deal with color constancy, 
specially the lateral geniculate nucleus (LGN), and the regions VI and V4 of the visual cortex [1], 
although recent studies suggest that other areas might also be involved [8]. Therefore, as Smithson 
says [2] "It seems most reasonable to say that processing for color constancy starts in the retina, is 
enhanced in V1N2 and continues in V4". Cues for color constancy used by humans that might befall in 
the further neural levels might include mutual reflections, 3D shapes, shadows, color memory, and even 
the consciousness of illumination change [2,9]. 

1.2. Computational Color Constancy 

Computational color constancy does not aim to recover the perceived image, but an estimation of 
the surface reflectances of the scene; Therefore, it changes the paradigm upon which human color 
constancy is built. In other words, while human color constancy relies on the perception of the 
colors, computational color constancy relies on the absolute color values of the objects viewed under 
a canonical (usually white) illuminant, without considering how the image is perceived by an observer. 
Vazquez-Corral et al. [10] have shown that computational and human color constancy do not aim for the 
same final image: they performed a pair- wise comparison experiment where they asked human observes 
to pick out the most natural image from a range of images produced by different computational color 
constancy algorithms. Observers only chose the best computational solution in 40% of the comparisons. 

Therefore, computational color constancy is treated from a mathematical point of view. Let us 
suppose we have an object with reflectance R(X) and a camera with sensitivities Qi(X). We take two 
photos of the object with the camera at two different moments in time; at each of these moments there 
is a different illuminant in the scene -E' 1 (A), E 2 (X) (let us suppose the illuminant is uniform). Then, the 
response of the object recorded by the camera sensor i under one of the illuminants is denoted by pf 
where the superscript n denotes the illuminant used. Mathematically, 
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(5) 



Marimont and Wandell [11] showed that although natural surface reflectances in the world are 5 or 6 
dimensional, i.e., there is a need of a basis of 5 or 6 reflectances in order to derive any other reflectance, 
an "effective basis" of smaller dimension can be extracted. Similarly, Judd et al. [12] showed how to 
derive a basis from the set of daylight illuminations. These studies allow us to relate p 1 = [p\, pi, p\] 
and p 2 = [p\, p\, /?§] by a 3 X 3 matrix. That means, we are dealing with a 9-dimensional problem: 



Equation (6) is crucial in order to solve the computational color constancy problem. If we wish 
to estimate reflectances by discounting the color of the prevailing light from the picture of a scene, it 
suffices to find the matrix that replaces this light by the canonical one. As an example, if an image is 
captured under bluish light then all the recorded sensor responses are biased in the blue direction and, 
in particular, a white surface will itself be bluish. If we can find the matrix that takes us from the blue 
light to a white counterpart then applying this matrix will remove the colored light. Of course, as simple 
as Equation (6) is, there are 9 components in a 3 x 3 matrix and so color constancy, viewed in this 
perspective, is a hard 9-dimensional problem. 

2. Spectral Sharpening for Diagonal Color Constancy 

Early research in computational color constancy [5,6] focused on modeling illumination changes by a 
scaling multiplication in each of the sensor channels (inspired by von Kries' coefficient law). This idea, 
called the diagonal model of illuminant change, can be expressed as: 



where D 1 ' 2 is a 3 x 3 diagonal matrix. Equation (7) supposes that illumination change is a process 
that operates in each sensor channel independently, which simplifies color constancy computation. The 
diagonal model turns out to be rather good at accounting for illuminant change in many circumstances. 
Partly, this is explained by the underlying physics which states that as the support of the sensor becomes 
small, i.e., as the range of wavelengths to which the sensor responds becomes smaller, then a diagonal 
matrix will work well [13]. Empirically, it has been shown that a diagonal matrix works for most cameras 
that have spectral sensitivities with support of 100 to 150 nanometers [14]. Let us in any case note that 
this is different from von Kries adaptation, which should be performed in the L,M,S cone space. 

Many cameras do not have sensors that match the above specifications. Therefore, different methods 
have tried to search for a linear combination of the original sensor responses in order to force them to 
accomplish the diagonal model. Mathematically, this linear combination will be the one accomplishing 



p 1 = M^ 2 p 2 . 



(6) 



p 1 « D 1,2 p 2 



(7) 



Tp 1 « D 1,2 Tp 2 



(8) 



Finlayson et al. [15] called this approach "spectral sharpening" since the new sensors responses are 
sharper than the original ones. We should note that other authors had previously suggested a similar 
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idea [16]. Figure 2 compares a set of originals sensors with their sharpened version. The remarkable 
and useful conclusion of the spectral sharpening work was that, even for a broad-band sensor system, a 
diagonal matrix model of illumination could be used to solve the computational color constancy problem. 

Figure 2. Original camera sensors (left) and their sharpened counterparts (right). 
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An example of the use of spectral sharpening can be seen in Figure 3. In this figure, an original image 
that presents a blue cast is seen under the CMF color matching functions. In order to apply diagonal color 
constancy and remove the blue cast, a linear transform T is computed by 5 different methods (that will 
be explained later in this section). Then, for each method, the pipeline works as follows: (1) a change 
of basis is performed by the linear transform T; (2) a method for diagonal color constancy (MaxRGB in 
this case) is applied to the new basis image; (3) the image is converted back to the original sensors by 
T _1 . The different T matrices have been obtained using the Planckian illuminants and the whole set of 
reflectances from [17]. 

In this section we will show a review of different methods used to achieve spectral sharpening. 
Figure 4 presents a hierarchy regarding when each particular method might be used. In this figure, the 
different methods are linked to their section in the paper. The selection of a particular method might be 
taken depending on two aspects: the availability of spectral data and the final goal that is being pursued. 

2.1. Perfect Sharpening 

Finlayson et al. [15,19] showed that when illuminants are two dimensional (in the sense that two 
illuminants are enough to define any other illuminant as a linear combination of them) and reflectances 
three dimensional (in the same sense), or vice versa, spectral sharpening is perfect. 

Let us suppose that reflectances are three dimensional and illuminants two dimensional (the other 
case is analogous). In this case any reflectance can be decomposed as 

3 

R(\) = y £n j {\)<T j (9) 

3=1 

where TZj(X) is a basis and a = \<j\, <7 2 , cr 3 ] is a coefficient vector in this basis. Let us define A fc as a 
3x3 matrix which ijth entry is defined as Ay = J u Qi(\)E k (\)1Zj(X), where superscript k denotes the 
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illuminant used and Qi are the sensors to be sharpened. Then, a color descriptor (Equation (5)) under a 
canonical light c can be written as 

f = A c a (10) 



Figure 3. Example of diagonal color constancy using spectral sharpening for five different 
methods. The original image (sensors) is converted by a linear matrix T to a sharpened 
basis. Then, a diagonal color constancy method is applied (MaxRGB in this case). Finally, 
the resultant image is converted back to the original basis by the inverse of the matrix T. In 
this example, the original sensors are the CMF XYZ functions. The sharpening matrices have 
been obtained using the Planckian illuminants and the whole set of reflectances from [17]. 
The multispectral image comes from [18]. 
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As illuminants are two dimensional they need a second illuminant E 2 {X) independent from the 
canonical one E c (\) to span the space. Associated with this illuminant there will also be a new lighting 
matrix A 2 . This second lighting matrix is some linear transform (M) away from the first one, A 2 = MA C , 
that is, M = A 2 [A C ]- X . 

As E 2 (X) and E c {\) span the space, any other lighting matrix will be a combination of A c and MA°. 
For this reason, any color descriptor under an illuminant E e {\) = aE c (X) + (3E 2 {\) can be written as 



f = [ a l + (5M]k c a = [a I + f3M}p c 
where I is the identity matrix. Calculating the eigenvector decomposition of M 

M = T~ X DT 



(11) 



(12) 
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and expressing the identity matrix in terms of T, / = T 1 IT, they rewrite Equation (1 1) as a diagonal 
transform 



Finally, writing p c in terms of p e 



Tp e = [al + f3D]Tp c 



(13) 



Tp c = [al + pD}~ 1 Tp e . 
This implies that the spectral sharpening is perfect since both matrices D and / are diagonal. 



(14) 



Figure 4. Hierarchy for the selection of a spectral sharpening method. The decision should 
take into account two aspects: the final goal pursued and the availability of spectral data. 
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2.2. Sensor-Based Sharpening 

Finlayson et al. proposed in [15] a method called Sensor-based spectral sharpening. The idea 
underlying this method is that it is possible to sharpen a sensor from an original set Q(X) (of dimension 
n x k) in a wavelength interval [Ai, A 2 ]. The resulting sensor Q(X)t where t is a coefficient vector of 
dimension k, can be found by minimizing 

min ^[Q(A)t] 2 + fi{ f [Q(X)t] 2 d\ - 1} (15) 

where u is the visible spectrum, $ denotes wavelengths outside [Ai, A 2 ] and p, is a Lagrange multiplier. 
In other words, the idea is to strengthen the percentage of the norm of sensor Q(X)t lying in the interval 
[Ai, A 2 ] in relation to the rest of the spectrum. 
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To solve the problem for all the spectra Finlayson and co-authors defined k intervals, where k is the 
number of sensors. These intervals have no intersection between them and cover all the spectra. Then, 
the kth row of matrix T will be the vector that minimizes Equation (15) for its particular interval (note 
that t is post-multiplying in Equation (15) while T is defined in Equation (8) as a pre-multiplication). 

Mathematically, they defined a k x k matrix 



where A a is an operator that picks out wavelengths indices in the sharpening interval a within any sum. 

They took partial derivatives over the vector t in Equation (15) and equated to the zero vector to look 
for the stationary values. They combined this derivative with A(ct) obtaining: 



In parallel, they differentiated Equation (15) over fi and found the constraint ^AecJ^Mtl 2 = 1- 
Rearranging Equation (17), they concluded that finding t amounts to solving eigenvector problem 



Then, as this last equation has multiple solutions, they choose one solution minimizing ^2\ e ^[Q(^)t\ 2 - 

2.2.1. Sharpening with Positivity 

Sensors with negative values are physically impossible. For this reason Pearson and Yule [20] 
defined different positive combinations of the color matching functions. Following this trend, Drew 
and Finlayson proposed methods to obtain sharpening transforms giving always positive values [21]. 
These techniques are very similar to the previous one but some constraints were added for ensuring all 
the values are positive. These constraints can be based either on the L\ or L 2 norm and can be performed 
both in the sensors themselves or in the sharpening matrix coefficients. All these methods can be solved 
by either linear or quadratic programming. Here we report the different methods presented. 

• L\- L\ Constrained coefficients: 




(16) 



A($)t + fiA(u)t = 0. 



(17) 



A(tu)- 1 A($)t = -fit. 



(18) 




(19) 



• L\-L\ Constrained sensors: 




(20) 



• L 2 - L 2 Constrained coefficients: 




(21) 
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L 2 - L 2 Constrained sensors 



argminV[g(A)t] 2 constrained to { mn EJW)t| 2 1 (22) 

L 2 - Li Constrained coefficients 

argmin^[Q(A)t] 2 constrained to I mn £ w [Q(A)tl = 1 (23) 



• L 2 - Li Constrained sensors 

argmin V[Q(A)t] 2 constrained to { ™ n = 1 (24) 

* $ [ g(A)t > o 

2.3. Adding Information to Improve Sharpening 

Information about the illuminants and reflectances that are more representative in natural scenes is 
available from multiple sources [12,17,22-26]. In this section we review methods that take advantage of 
this available information to search for sharpened sensors. 

2.3.1. Data-Based Sharpening 

Finlayson et al. [15] proposed a method called data-based sharpening which uses linear algebra 
methods to directly solve for T by minimizing the residual error between a pair of illuminants. To this 
end, they defined W l and W 2 as 3 x n matrices containing the color values for a set on n different 
refectances under two different illuminants E 1 and E 2 . 

TW 1 ss D 1 ' 2 TW 2 . (25) 

Then, they solved Equation (25) for D 1 ' 2 in a least-squares sense. This can be done by the Moore-Penrose 
inverse 

D 1 ' 2 = TW 1 [TW 2 ] + (26) 
where [] + represents the pseudoinverse [27]. Rearranging Equation (26), they got 

T~ 1 D 1 > 2 T = W 1 [W 2 ] + . (27) 

Therefore, T is the eigenvector decomposition of W 1 [W 2 ] + . 

Some years later, Barnard et al. [28] tried to allow more flexibility to the database sharpening, by 
averaging over a set of illuminants (not only one), and introducing a parameter to prioritize positivity. 

2.3.2. Measurement Tensor 

Chong et al. [29] introduced a new method which finds a matrix T for a complete set of illuminants 
at the same time. This method is based on the measurement tensor defined as 

M kij := / Q fc (A)^(A)^(A)dA (28) 
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where {Ei} i=1 1 is a set of illuminants, {Rj}j =l ... j is a set of reflectances and {Qk\k=i- K are 
sensors. This measurement tensor is an order 3 tensor. 

Chong et al. proved that a measurement tensor supports diagonal color constancy if and only if it is 
a rank 3 tensor. An order 3 tensor r is rank iV if iV is the smallest integer such that there exist vectors 
{a n , b n , c n } n= i 5 ... t N allowing decomposition as the sum of outer products 

N 

T = ^9n° On°bn (29) 
n=l 

where o represents the outer product. 
They rewrote Equation (29) as 

TV 

r = J2 C ° Ao B (3°) 

n=l 

Where the columns of A, B and C are composed by the different a n , b n , and c n respectively. C is the 
matrix we search, and T = C -1 . 

In order to solve Equation (30) they used the Trilinear Alternate Least Squares (TALS) method [30]. 
This is necessary since in most of the cases the tensor M ki j is not rank 3, and therefore it is necessary 
to search for the "closest" rank 3 tensor. At each iteration of the minimization procedure through TALS, 
two of the three matrices are fixed while the free matrix is chosen to minimize the difference between the 
given data M kij and the obtained tensor r in the least-squares sense. The alternating process is repeated 
until convergence. 

This method has the drawback of local convergence, that is, the result obtained can be a local minima. 
Also, TALS needs initialization values for two of their three matrices. 

2.3.3. Data-Driven Positivity 

In [21], Drew and Finlayson proposed a data-driven approach for obtaining positive sensors. 
Following the original sensor-based sharpening method they divided the spectra in k intervals, where 
k is the number of sensors, and for each interval they searched for the sensor Q(\)t minimizing 

min£jQ(A)r = l 
Rt > 0 

where $ denotes the wavelengths outside the selected interval , cu represents the visible interval, R is 
a r x 3 matrix representing the gamut boundary of the set of RGBs obtained from the data, r represents 
the number of points lying on that boundary, and v = 1,2 represents the chosen norm. The transpose 
of the vector t is the k-th row of the sharpening matrix T. This method does not guarantee per se the 
positiveness of the result. Positiveness is conditioned to select a big enough space of reflectances and 
illuminants so no other color signal lies out of the gamut defined by R. 

2.4. Chromatic Adaptation Transforms 

All the previous methods were defined in order to help solving for diagonal color constancy. But, 
the sharpening matrices related to these methods are not the only ones, there are sharpening matrices 



min ^[Q(A)t] 1 ' constrained to 
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that have been derived from psychophysical experiments in chromatic adaptation. Chromatic adaptation 
matrices also represent sharp sensors [31], and are obtained from the XYZ color matching functions. 
They are used to match image appearance to colorimetry when the visual conditions are changed. In 
particular, they are defined to handle corresponding colors data. Citing from the Fairchild's book [32] 
"Corresponding colors are defined as two stimuli, viewed under different viewing conditions, that match 
in color appearance" . Examples of chromatic adaptation transforms are the Bradford transform, the 
Fairchild transform, and the CAT02 transform. For a review on the different transforms, we recommend 
the book by Fairchild [32]. Here we enumerate some of them. 

2.4.1. Von Kries Transform 

The Von Kries chromatic adaptation transform is usually defined by the Hunt, Poynton and Estevez 
transform [32]. The values of this transform are: 



/ 0.3897 0.6890 
-0.2298 1.1834 
0 



0 



-0.0787 \ 
0.0464 

1 / 



(32) 



2.4.2. Bradford Transform 



The Bradform transform was defined by Lam [33] following the results obtained in an experiment 
regarding corresponding colors. The data used for the experiment consisted of 58 dyed wood samples 
under the A and D65 illuminants. The original Bradford transform is non-linear, but the non-linear part 
is usually neglected. The linear matrix is then 



/ 0.8951 
-0.7502 
\ 0.0389 



0.2664 -0.1614 \ 
1.7135 0.0367 
-0.0685 1.0296 j 



(33) 



2.4.3. Fairchild Transform 



The Fairchild transform was suggested by Mark Fairchild [34] for improving the CIECAM97s 
color appearance model. It was obtained through a linearization of the previous chromatic adaptation 
transform. The matrix suggested by Fairchild was 



T 



( 0.8562 
-0.8360 
\ 0.0357 



0.3372 
1.8327 
-0.0469 



-0.1934 \ 
0.0033 
1.0112 j 



(34) 



2.4.4. CAT02 Transform 



The Comission Internationale de VEclairage (CIE) selected in its report CIC-TC8-01 the CAT02 
transform as the preferred chromatic adaptation transform. CAT02 was obtained by optimizing a wide 
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variety of corresponding data, while approximating the non-linear transformation of CIECAM97s. The 
matrix obtained was 

/ 0.7328 0.4296 -0.1624 \ 
T= -0.7036 1.6975 0.0061 (35) 
\ 0.0030 0.0136 0.9834 j 

2.4.5. Chromatic Adaptation Transforms by Numerical Optimization 

Bianco and Schettini [35] defined two new chromatic adaptation transforms based on estimating the 
corresponding colors data from [4,33,36^-0]. 

They defined an objective function with two competing terms 

fBs(T) = g es t{T) - g me d{T). (36) 

The term g est becomes bigger for better estimations of corresponding colors according to the 
Wilcoxon signed-rank test. On the other hand, the term g med becomes smaller when the median errors on 
the corresponding colors datasets are smaller. Therefore, our goal must be to look for the transformation 
T maximizing Equation (36). To this end, authors applied the Particle Swarm Optimization (PSO) 
technique. 

This first optimization might, however, incur in negative values of the resulting sensors. For this 
reason they defined another objective function 

fBs-pc(T) = 9est{T) - g me d(T) + gpc(T). (37) 

In this second function they added a positive competing term gpc that prioritizes positivity of the sensors. 
The second transform was obtained by maximizing Equation (37) though PSO. 



2.5. Spherical Sampling 

Spherical sampling [31,41] provides a means for discretely sampling points on a sphere and relate 
them to sensors. The main idea is to consider each row of the sharpening matrix T as a point in the 
sphere. 

Mathematically, let us represent our original camera sensors Q as a m x 3 matrix where m is the 
wavelength sampling and 3 the number of sensors. We perform the reduced singular value decomposition 
(SVD) of these sensors in order to obtain a basis: 

Q = U • £ • V* (38) 

where U is an orthogonal matrix with dimension m x 3, S is a diagonal 3x3 matrix containing the 
singular values of matrix Q and V* is an orthogonal 3x3 matrix. Then, U is the basis we seek. 

From this basis U, we can define a new set of sensors S (m x 3), different from the original sensors 
S, by multiplying the basis by any linear transformation P (3 x 3), which simply consists of 3 sample 
points vectors, pi, • • • ,P3 located over the 2-sphere. Then, 



Q = UP, P=[pi,---,y 



(39) 
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We are interested in the relation between the original sensors Q and the new defined ones Q. Using 
Equations (38) and (39) we find 

Q = UP = UZV t (Y;V t )~ 1 P = g(SV r *)" 1 P (40) 

Therefore, relating this equation to Equation (8) where T is pre-multiplying we obtain 

T = ((EV t )- 1 P) t (41) 

We can also rearrange this equation in order to relate a transformation matrix T with a set of points P 
over the sphere. 

P = EV^T* (42) 

2.6. Measuring Diagonal Color Constancy Effectiveness 

Effectiveness of sharpening matrices has usually been evaluated by least-squares as follows. Let us 
denote an observed color by p E (Equation (5)) where E is the illuminant and r the reflectance for the 
observation. Then, if we select a canonical reflectance s (usually an achromatic reflectance), we can 
compute for each illuminant the ratio between any reflectance and the white reflectance as follows. 



dl = T-\diag{Tp E s )]^Tg (43) 

where d E s is a vector of dimension 3, 

Let us note that if the transformation T perfectly accomplishes diagonal color constancy, the value 
d E s is independent from the illuminant. Therefore, measuring the disparity of this ratio depending on the 
illuminant should tell us the effectiveness of a method. 

Mathematically, if we select a canonical illuminant E°, we can denote the error of the sharpening 
matrix by 

\\d EC -d E II 

Error = 100 x "~ r,a ~ (44) 
\d II 

\\^r,s II 

This formula has been widely used to compare spectral sharpening methods and was already included 
in Finlayson et al. [15] work. By using this formula two methods outperform the rest. First, the 
Measurement Tensor method as shown in [29]. This method has an inherent advantage because 
both the method and the measure are based on least- squares. Therefore, we deal with a least square 
minimization-least square evaluation paradigm. The second method that excels is Spherical Sampling 
due to its capability to minimize any measure. Spherical Sampling presents a further advantage since it 
avoids local minima. 

The formula presented in Equation (44) is good for a first inspection on how the methods work with 
simple diagonal color constancy. But, recently, further applications of sharpened sensors have been 
found (see next section) where this measure is no longer appropriate. 
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3. Beyond Diagonal Color Constancy 

The original aim of spectral sharpening was to achieve diagonal color constancy. Over the years, 
spectral sharpening has proven beneficial for a number of purposes, some far removed from the original 
aim. In this section we review some of these new applications. They are presented graphically in Figure 5 
where they are listed in terms of their research field and linked to a particular subsection of this paper. 

Figure 5. Hierarchy of sensor sharpening applications grouped by research field. Each 
application is linked to a section in this paper. 



Sensor sharpening 
applications 



Colorimetry 



Perceptual image 
processing 



Image processing 
and 

computer vision 



Estimating human 
color properties 



Diagonal color constancy 
(Section 2.6) 



Chromatic adaptation 
(Sections 2.4 and 3.1) 



Color constancy in perceptual 
spaces 
(Section 3.2) 



Relational color constancy 
(Section 3.3) 



A perceptual-based color space 
(Section 3.4) 



Multispectral processing 
without the use of all spectra 
(Section 3.5) 



Invariant images and shadows 
(Section 3.6) 



Information estimation from 
image colors 
(Section 3.7) 



Color names, unique hues, hue 
equilibrium and hue 
cancellation prediction 
(Section 3.8) 



3.1. Chromatic Adaptation 

Section 2.4 shows that chromatic adaptation transforms can be understood as spectral sharpening, 
therefore it is a straightforward idea to use spectral sharpening techniques for handling corresponding 
colors data and chromatic adaptation. 

Finlayson and Drew [42] showed that the Bradford transform can be obtained through spectral 
sharpening with a careful selection of intervals. Later on, Finlayson and Siisstrunk in [43] defined 
a chromatic adaptation transform following a technique very similar to the data-based sharpening 
considering the preservation of the white point. Ciurea and Funt [44] used the same algorithm but 
applied it to spectral quantities instead of tristimulus values. Finally, Finlayson and Siisstrunk [31] used 
the spherical sampling technique to derive a set of chromatic adaptation transforms that were equivalent 
in terms of the error committed to the colorimetrically obtained. 
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3.2. Color Constancy in Perceptual Spaces 

Human perception is not linear but colorimetric spaces are. In other words, when we work in RGB or 
XYZ spaces, a Euclidean distance d will be perceived differently depending on the region of the color 
space the points are located in. 

To overcome this issue CIE proposed the CIELab and CieLuv color spaces [45]. Later on, 
Finlayson et al. [41] defined a new color constancy error measure regarding differences in the CIELab 
perceptual space. From Equation (5), let us call p m5 the XYZ color value of a particular patch under the 
D65 illuminant and p e the value of the same patch under a different illuminant e. We know we can find 
an approximation of the value under the DQ5 illuminant by p m5 = T _1 • D ■ T ■ p e . The basic idea is 
to convert both p D65 and p mb values to CIELab and to minimize the measure A e , that is, the euclidean 
distance between the two points. This measure is considered to be perceptual. Formally, 

A e (T) = \\Lab(p mB ) - Lab(p m5 )\\ = \\Lab(p m5 ) - Lab(T~ l ■ D ■ T ■ p e )\\ (45) 

The matrix T minimizing this equation for a set of reflectances and illuminants is defined as the best 
matrix regarding perceptual color constancy. It is found using the spherical sampling technique [41]. 

3.3. Relational Color Constancy 

Foster and co-authors [46,47] defined color constancy as a ratio-based phenomena, not pixel-based. 
This view, called relational color constancy, assumes that the colors in a scene have a fixed relation 
between each other. Relational color constancy is also related to Retinex [48]. On the other hand, from 
the computer vision side, color ratios have proven useful for dealing with some particular problems such 
as object recognition [49] and image indexing [50]. 

Finlayson et al. [41] defined a color ratio stability measure that works in each sensor individually. 
They defined a vector b (m-by-1) containing the colors for a set of m reflectances under the canonical 
illuminant viewed under a particular sensor, that is b = [Ci, • • • , Cm] — [(Tp^)i, ■ ■ • , (Tp c ),], where 
Tp° is the response of the sensors for reflectance m and canonical illuminant c, and subscript i denotes 
the sensor selected. They defined the vector of color ratios a c 

GXjCb, Q^Q (46) 

They considered a second vector of ratios for the same reflectances under a different illuminant a e . In 
this case, the total ratio error is defined by 

^m^'-t^f <47) 

This error is minimized by the spherical sampling method [41]. 

3.4. A Perceptual-Based Color Space for Image Segmentation and Poisson Editing 

Chong et al. [51] defined a perception-based color space with two main goals: (1) be linear correlated 
with perceived distances, that is, distances in this space might correlate with perceived ones, and (2) 
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color displacements in this space should be robust to spectral changes in illumination, that is, if we 
re-illuminate two colors by the same light, the difference between them might stay equal. To obtain 
a space with these characteristics the authors showed that one further assumption is needed. This 
assumption states that diagonal color constancy must be well modeled in the space. 

Therefore, the definition of the color space parametrization F given a point x in XYZ coordinates is 

F(x) = A(ln(Bx)) (48) 

where B is coding a change in a color basis by converting original sensors into ones where diagonal color 
constancy is well characterized, In is the natural logarithm and A is used to match perceptual distances. 
To obtain B authors used the measurement tensor method [29]. 

The authors showed the advantages of this new space in two common image processing tasks: image 
segmentation and Poisson editing [52]. 

3.5. Multispectral Processing Without Spectra 

Drew and Finlayson [53] showed that spectral sharpening was useful to simplify the cost of calculating 
the modeling of the interaction of light and reflectance. Reducing this cost is important for problems such 
as ray-tracing. For this application they applied spectral sharpening in more than the usual 3 dimensions 
(red, green, and blue). 

First, they defined a set of color signals, from which they obtained a n-dimensional (n = 5, 6, 7) basis 
B(X) (via SVD). They sharpened this basis to improve the discernability of its information obtaining a 
new basis B(\) = TB(X), where T is a n x n sharpening matrix obtaining using the method L 2 — L 2 
sensor-based with positivity. Then, any light or reflectance can be expressed as a coefficient vector in 
this last basis 

N 

R(\) = J2l>M\) (49) 

i=l 

Drew and Finlayson showed that computing the modeling of light and reflectance using these coefficient 
vectors and then reconstructing back the full color signal needs less computations and that the error 
committed is very small. 

Later on, Finlayson et al. [41] showed that even smaller errors are obtained by the use of spherical 
sampling in terms of the A e measure between the Lab values of the real and the reconstructed signal. 

3.6. Obtaining an Invariant Image and Its Application to Shadows 

Finlayson et al. [54] theoretically proved that it is possible to obtain a 1 -dimensional representation 
of reflectances independent from the illuminant if one supposes a narrow-band camera. From this 
1 -dimension representation they obtained the invariant image, where the RGB value of the pixel is 
substituted by its illuminant-independent representation. In the same work they also showed that the 
illuminant independent representation was useful in real cameras (the narrower the camera sensors, the 
better the results). Following this last point, Drew et al. [55] proved that when using spectral sharpening 
sensors, the invariant image was better than with the original ones. 
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Finlayson et al. [56], later used the invariant image to remove shadows from images. Drew et al. [57] 
recently showed that when using spectral sharpening sensors, the results were improved, although this 
last algorithm requires user interaction. 

3.7. Estimating the Information from Image Colors 

Recently, Marin-Franch and Foster [58] presented a method to evaluate the amount of information 
that can be estimated from the image colors captured by a camera under different illuminations. To this 
end, they applied different statistics to the color images. In their work they explained that when dealing 
with spectrally sharpened sensors the amount of information that can be extracted is higher than with the 
original ones. 

3.8. Color Names, Unique Hues, Hue Cancellation and Hue Equilibrium 

Philipona and O'Regan [59] showed the possibility of extracting a surface reflectance descriptor using 
only the information reaching our eye. To start with, they defined v s as the accessible information about 
the reflected light for a given surface s and u the accessible information about the incident illuminant 

Q i {\)E{\)R{\)d\ z = 1,2,3 (50) 
u = / Qi(\)E{\)d\, i = 1,2,3 (51) 

J 10 

where R(X), E(\) are the same as in Equation (5) and Qi(X) is the absorption of photopigments 
presented in the L,M and S photoreceptors. 

They repeated this procedure for iV different lights. They arranged the N response vectors for the 
surface in a 3 x iV matrix V s and for the lights in a 3 x N matrix U . They related these two matrices by 
finding the best 3x3 matrix transform A s such that 

V s ~ A s ■ U (52) 

where the superscript s denotes dependence on the surface. They solved for the matrix A s by linear 
regression. Finally, they considered the eigenvalue/eigenvector decomposition of A s reaching to 

V s ~U S V S (W)- 1 ■ U (53) 

where W and V s are the 3x3 matrices of the eigenvectors and eigenvalues respectively. 

Philipona and O'Regan selected the eigenvalues V s as the surface descriptor, from where they were 
able to precisely predict color naming, unique hues, hue equilibrium and hue cancellation data. 

Building upon Philipona and O'Regan's model, Vazquez-Corral et al. [60] demonstrated that it is 
possible to find a unique transformation T for all the surfaces such that the Philipona and O'Regan 
model can be expressed as 

v s ~ TV S T~ X ■ u (54) 

This transformation T was computed by spherical sampling. The work of Vazquez-Corral et al. shows 
that the surface reflectance descriptor is equivalent to a Land color designator [6] computed in the space 
spanned by the sharp sensors. 
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4. Conclusion 

Sensor sharpening was developed 20 years ago to achieve computational color constancy using a 
diagonal model. Over this period, spectral sharpening has proven important for solving other problems 
unrelated to its original aim. 

In this paper, we have explained some of the differences between human and computational color 
constancy: human color constancy relies on the perception of the colors while computational color 
constancy relies on the absolute color values of the objects viewed under a canonical illuminant. 

We have reviewed different methods used to obtain spectrally sharpened sensors, dividing them into 
perfect sharpening, sensor-based sharpening, sharpening with data, spherical sharpening, and chromatic 
adaptation transforms. 

We have also described different research lines where sharpened sensors have proven useful: 
chromatic adaptation, color constancy in perceptual spaces, relational color constancy, perceptual-based 
definition of color spaces, multispectral processing without the use of all the spectra, shadow removal, 
extraction of information from an image, estimation of the color names and unique hues presented in the 
human visual system, and estimation of the hue cancellation and hue equilibrium phenomena. 
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